Skip to main content

Work with Typesense

All workflows and schemas related to typesense are maintained in visn-devops-scripts repository.

Schema and data-design guidelines​

Typesense is a search projection, not the source of truth. Keep the canonical record and its complete data structure in Firestore; put only the fields required to find, filter, rank, sort, group, or identify that record in Typesense. This keeps documents small and avoids data becoming inconsistent when Firestore data changes.

Before adding a field​

Decide how the application will use the field before adding it to a Typesense schema.

Application needRecommended approach
Identify the record returned by searchStore the Firestore document ID and any IDs needed to construct its Firestore path.
Full-text search or matchingAdd the field to Typesense and index it. Use the appropriate scalar or array type.
Display-only dataPrefer loading it from Firestore after search. Do not add it to Typesense simply to render it.

Data structure and field types​

  • Model the Typesense document as a flat, query-focused projection of the Firestore document. Do not copy large nested objects, histories, logs, or unused arrays into the search document. For example, if the application only needs to search by the types of enabled services, reduce the Firestore objects to a simple array in Typesense:

    // Firestore
    services: [
      { type: 'SERVICE', enabled: true },
      { type: 'MOT', enabled: false },
    ];

    // Typesense
    enabledServiceTypes: ['SERVICE'];

  • Prefer stable primitive values: string, int32/int64, float, bool, and correctly typed arrays. Use arrays only when one record genuinely has multiple searchable/filterable values.

  • Use consistent representations across every document. For example, do not mix numbers and numeric strings, booleans and "true"/"false", or a scalar and an array for the same field.

  • Use an optional field when older records or valid use cases may not supply a value. Do not use placeholder text such as "N/A" solely to satisfy a schema; missing data should remain missing where possible.

  • Denormalise only values needed to make the search query work. When a Firestore value is copied into Typesense, ensure the indexing/update path updates it whenever the Firestore value changes. For example, if users must search enquiries by lease company name, copy that name into each enquiry search document and re-index the affected enquiries when the company name changes:

    // Firestore: /leasecompany/{leaseCompanyId}
    { name: 'Acme Leasing' }

    // Typesense: enquiry search document
    { enquiryId: 'ENQ-123', leaseCompanyName: 'Acme Leasing' }

    // When the Firestore name changes to 'Acme Fleet', update every
    // related enquiry search document with: { leaseCompanyName: 'Acme Fleet' }

  • Consider privacy and access control before indexing: Typesense documents can be returned in search responses. Do not include sensitive or user-specific fields unless the collection/query access controls are designed for them.

Creating a new collection​

Keep the schema minimal​

Keep the schema basic: add only the fields required for search, filtering, sorting, faceting, or identifying the Firestore record. Do not add fields simply because they are available in the Firestore document.

For display-only data, prefer retrieving the data from Firestore after Typesense returns the matching record IDs. This keeps Firestore as the source of truth and avoids duplicating data that is not needed to perform the search.

Field naming​

Prefix every field in a new collection with a unique, collection-specific prefix. For example, fields in accountsIndex should use the ai_ prefix:

ai_accountId
ai_name
ai_status

This is required because scoped search keys use include_fields to control which fields may be returned in a response. Without the prefix, a common field name such as name can exist in more than one collection. Adding name to include_fields for the collection that needs it can then cause name to be returned from other collections too, even where it should not be exposed.

Choose the prefix before creating the schema, apply it consistently to every field in the collection, and use the same prefixed names in import code, queries, and scoped search-key configuration. For example, add ai_name—not name—to include_fields when the account name is allowed in the response.

When to reindex dependent collections​

Typesense stores the internal ID of the referenced document in each dependent document. As these IDs are assigned during indexing, joined collections must be treated as one reindexing group when a referenced collection is rebuilt and switched through an alias.

For example, the following collections reference subCustomerIndex:

subCustomerIndex
├── enquirySearchIndex
├── enquiryStatusIndex
└── stockIndex

Note: These collections are examples only. Always check the relevant Typesense schemas to identify every collection that references the rebuilt collection before planning a reindex. All schemas are in src/schemas/typesense in the visn-devops-scripts repository.

  1. When subCustomerIndex is rebuilt: rebuild and reindex subCustomerIndex first. Then reindex enquirySearchIndex, enquiryStatusIndex, and stockIndex. Each dependent collection must be reindexed so that it stores the internal IDs from the rebuilt sub-customer collection. Coordinate the alias updates for this whole collection group.

  2. When only a dependent collection is rebuilt: reindex only that collection. For example, if stockIndex has a schema change, reindex stockIndex only. subCustomerIndex, enquirySearchIndex, and enquiryStatusIndex do not need to be reindexed.

  3. When a reference definition changes: reindex each dependent collection that uses the changed reference. For example, if reference: 'subCustomerIndex.subCustomerId' changes, rebuild every collection that contains that reference field.

  4. When only a document is updated in the current collection: update that document normally. A full dependent-collection reindex is only needed when a referenced collection is rebuilt/switched, or when dependent documents contain copied (denormalised) values that also need refreshing.

For more detail, see the Typesense documentation on using aliases with joins.

How to update or add a field in existing schema​

Before changing the schema, confirm the field's purpose, type, optionality, indexing requirement, and facet requirement using the guidance above. Also identify every writer that imports documents and every client/query that consumes the field.

  1. First update or add the field in the schema in code
  2. Use the Create collection workflow to create a new collection with updated schema
  3. Now use the Migrate to new collection workflow to migrate the data from old collection to new collection
  4. After all data is migrated successfully, use the update alias workflow to update the alias to point to new collection
  5. After successfully updating the alias, use the delete collection workflow to delete the old collection

Creating a collection workflow​

Workflow Used: .github\workflows\DEV-create-typesense-collection.yaml

  1. Create a schema file for that index under src\schemas\typesense ( More about how to create schema can be found in references )
  2. Add the newly created schema in src\schemas\typesenseSchemaLookUp.ts (Keep the typesenseCollectionsConstant same as alias name)
  3. Next need to trigger the above mentioned workflow, with inputs as Collection name: Should be same as alias that is used for schema Suffix: Any name for example date (aug_2023) or release_11_0_0(release is recommended for higher environments)
  1. After successfully workflow run a collection will be created. Collection will follow collectionName_suffix format Eg. Collection Name: enquirySearchIndex Suffix: release_11_0_0 Final collection name: enquirySearchIndex_release_11_0_0

Note: if creating a new collection from existing schema then Step 1 & 2 can be skipped

Migrating data to new collection​

Workflow used: .github\workflows\DEV-migrate-to-new-collection.yaml

This workflow is used when we have updated a schema and created a new collection with updated schema, and want to move the data form the old collection to the newly created collection.

Note: Recommended to create a backup before running this workflow.

  1. Need to run the above mentioned workflow with following inputs. Alias: Alias name that is used for old collection (Same as collection name that was used during creating a collection) Suffix: suffix that was used when creating the new collection
  1. After this workflow runs successfully, need to verify that there are no failed documents to import( Refer the below screenshot)
  1. If there are failed documents need to fix those errors and re run the workflow.

Creating or updating an alias​

Workflow used: .github\workflows\DEV-update-typesense-alias.yaml

This workflow will be used to create or update an alias to point to new collection. You can read more about alias here.

  1. Use the above mentioned workflow with following inputs: collection: Full collection name, will be in the format alias_suffix (E.g. enquirySearchIndex_release_11_0_0) alias: The alias name the given collection should point to.
  1. After successfully running this workflow provided alias will be modified (or created if does not exits) pointing to the provided collection.

Deleting collection​

Workflow used: .github\workflows\DEV-delete-typesense-collection.yaml

This workflow will be used to delete a collection.

  1. Use the above mentioned workflow with following inputs: Collection: enquirySearchIndex_release_11_0_0 (full collection name should be provided)
  2. After successfully running this workflow the provided collection will be deleted

Note: if only the alias name is provided name as input, then the collection which the alias is pointing to will be deleted.

Backup​

Workflow used: .github\workflows\DEV-backup-typesense-data.yaml At the end of each day a scheduled workflow is run to create a backup of typesense data (Only in DEV and PROD environment).

References​

Status: Accepted
Category: Protected
Authored By: Gladson on Sep 25, 2023
Revisions.
15 Jul, 2026 - Add Typesense schema, collection naming, and join re-indexing guidance
- https://myvisn.atlassian.net/browse/VN-26898
Authored By: Gladson